Using Ecometric Data to Explore Sources of Cross-Site Impact Variance in Multi-Site Trials.

In 2003, Bloom, Hill, and Riccio (BHR) published an influential paper introducing novel methods for explaining the variation in local impacts observed in multi-site randomized control trials of socio-economic interventions in terms of site-level mediators. This paper seeks to improve upon this previous work by using student-level data to measure site-level mediators and confounders. Development of asymptotic behavior backed up with simulations and an empirical example. Students and training providers. Two simulations and an empirical application to data from an evaluation of the Health Professions Opportunity Grants (HPOG) Program. This empirical analysis involved roughly 6600 participants across 37 local sites. We examine bias and mean square error of estimates of mediation coefficients as well as the true coverage of nominal 95-percent confidence intervals on the mediation coefficients. Simulations suggest that the new methods generally improve the quality of inferences even when there is no confounding. Applying this methodology to the HPOG study shows that program-average FTE months of study by month six was a significant mediator of both career progress and long-term degree/credential receipt. Evaluators can robustify their BHR-style analyses by the use of the methods proposed here.

Robustness of ordinary least squares in randomized clinical trials.

There has been a series of occasional papers in this journal about semiparametric methods for robust covariate control in the analysis of clinical trials. These methods are fairly easy to apply on currently available computers, but standard software packages do not yet support these methods with easy option selections. Moreover, these methods can be difficult to explain to practitioners who have only a basic statistical education. There is also a somewhat neglected history demonstrating that ordinary least squares (OLS) is very robust to the types of outcome distribution features that have motivated the newer methods for robust covariate control. We review these two strands of literature and report on some new simulations that demonstrate the robustness of OLS to more extreme normality violations than previously explored. The new simulations involve two strongly leptokurtic outcomes: near-zero binary outcomes and zero-inflated gamma outcomes. Potential examples of such outcomes include, respectively, 5-year survival rates for stage IV cancer and healthcare claim amounts for rare conditions. We find that traditional OLS methods work very well down to very small sample sizes for such outcomes. Under some circumstances, OLS with robust standard errors work well with even smaller sample sizes. Given this literature review and our new simulations, we think that most researchers may comfortably continue using standard OLS software, preferably with the robust standard errors.

The Influence of the VERB campaign on children's physical activity in 2002 to 2006.

OBJECTIVES: We evaluated physical activity outcomes for children exposed to VERB, a campaign to encourage physical activity in children, across campaign years 2002 to 2006. METHODS: We examined the associations between exposure to VERB and (1) physical activity sessions (free time and organized) and (2) psychosocial outcomes (outcome expectations, self-efficacy, and social influences) for 3 nationally representative cohorts of children. Outcomes among adolescents aged 13 to 17 years (cohort 1, baseline) and children aged 9 to 13 years from cohorts 2 and 3 were analyzed for dose-response effects. Propensity scoring was used to control for confounding influences. RESULTS: Awareness of VERB remained high across campaign years. In 2006, reports of children aged 10 to 13 years being active on the day before the survey increased significantly as exposure to the campaign increased. Psychosocial outcomes showed dose-response associations. Effects lessened as children aged out of the campaign target age range (cohort 1, baseline), but dose-response associations persisted in 2006 for outcome expectations and free-time physical activity. CONCLUSIONS: VERB positively influenced children's physical activity outcomes. Campaign effects persisted as children grew into their adolescent years.

Methodology of the outcome evaluation of the VERB campaign.

This article summarizes the methods used in the outcome evaluation of the VERB campaign. The outcome evaluation was designed to measure the awareness and understanding of VERB among the target audience of children aged 9-13 years (tweens) and to determine the effect of VERB awareness on psychosocial and behavioral outcomes. Cohorts of tweens and parents were interviewed annually via a telephone survey (Youth Media Campaign Longitudinal Survey). The first cohort (baseline) was surveyed in 2002 prior to VERB advertising and was repeated annually through 2006. A second cohort was surveyed in 2004-2006. A third, cross-sectional sample was surveyed in 2006. Each cohort consisted of a nationally representative sample of tweens to enable generalizability to the nation as a whole. Propensity scoring was used to control for confounding influences. The outcomes were analyzed for dose-response effects (i.e., whether higher levels of awareness led to stronger effects) and overall awareness effects (i.e., the difference between tweens unaware of VERB and all tweens in the U.S.). Secular trends in tweens' physical activity during the life of the campaign were also examined. This article also discusses weighting and imputation, alternative analyses used to assess the adequacy of the propensity methods, and the challenges involved in media campaign evaluations.

Evaluation of a national physical activity intervention for children: VERB campaign, 2002-2004.

BACKGROUND: Amid concern for the consequences of physical inactivity among children, the Centers for Disease Control and Prevention started a campaign using commercial marketing methods to promote physical activity to children. DESIGN: Longitudinal study using a telephone survey to assess physical activity behaviors and attitudes at baseline and for 2 years of follow-up. Relationships of campaign awareness to behavioral and psychosocial effects were analyzed with use of propensity scoring. PARTICIPANTS: Nationally representative cohort of 2257 parent-child dyads. INTERVENTION: Marketing campaign (VERB) directed to all U.S. children aged 9 to 13 years. Components included general market and ethnic-specific advertisements on television and radio, in print, and through promotions in communities, schools, and on the Internet. Advertising ran nationally at consistent levels from June 2002 through June 2004. MAIN OUTCOME MEASURES: Psychosocial measures and self-reports of free-time and organized physical activity during nonschool hours in the week before the interview and on the day before the interview. RESULTS: After 2 years, a dose-response effect was detected in the study population. The more children who reported seeing VERB messages, the more physical activity they reported and the more positive their attitudes were about the benefits of being physically active. Children aware of VERB reported engaging in significantly more physical activity than children unaware of VERB. These results were considerably stronger than the effects after Year 1, which were only for physical activity among subpopulations. CONCLUSIONS: The VERB campaign continued to positively influence children's attitudes about physical activity and their physical activity behaviors and expanded the effects to more children. With adequate and sustained investment, health marketing shows promise to affect the attitudes and behavior of children.

Variable selection and raking in propensity scoring.

This paper discusses some practical issues in applying propensity scoring in the context of endpoint analysis in a pre-/posttest longitudinal design with an ordinal measure of treatment intensity and a high-dimensional potential covariate space: how many covariates to include in propensity models; how to evaluate the adequacy of tentative propensity models; and how to tailor models to provide hypercontrol on a limited subset of covariates. These issues arose in the evaluation of a health communication program.